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ASSESSING  THE  USE  OE  GAME-BASED  EXERCISES  IN  THE  STAEE 
ATTACK-THE-NETWORK  COURSE 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  increasingly  relies  on  serious  games  as  a  training  tool  for  tactical  tasks. 
The  use  of  game-based  exercises  is  nonetheless  a  novel  approach  for  training  human-intelligence 
tasks.  Human- intelligence  tasks  are  the  actions  related  to  collecting  information  from  people  and 
other  sources  (i.e.,  social  networks,  print,  and  visual  media)  to  identify  elements,  intentions, 
composition,  strength,  dispositions,  tactics,  equipment,  personnel,  and  capabilities.  In  order  to 
determine  the  extent  to  which  a  virtual  game-based  environment  provided  an  effective  means  to 
apply  human-intelligence  skills,  two  forms  of  scenario-based  practical  exercises  were  compared 
in  the  Attack- the-Network  (AtN)  course. 

Procedure: 

AtN  course  performance  and  perceptions  of  training  were  compared  across  students  who 
completed  traditional  paper-based  practical  exercises  and  students  who  completed  practical 
exercises  presented  in  the  Army’s  Enhanced  Dynamic  Geo-Social  Environment  game-based 
training  environment. 

bindings: 

The  game-based  practical  exercises  did  no  better  in  increasing  end-of-course  test  scores 
than  did  traditional  paper-based  practical  exercises.  In  addition,  the  paper-based  practical 
exercises  were  perceived  as  more  beneficial  to  learning  and  course  outcomes  compared  to  the 
game-based  practical  exercises.  These  results  add  to  the  growing  literature  that  fails  to  find  a 
relative  advantage  of  game-based  training. 

Utilization  and  Dissemination  of  bindings: 

The  findings  from  the  current  research  were  briefed  to  the  Maneuver  Center  of 
Excellence  Directorate  of  Training  and  Doctrine  and  to  the  U.S.  Army  Research  Eaboratory  - 
Human  Research  and  Engineering  Directorate  Simulation  and  Training  Technology  Center.  The 
findings  were  used  to  inform  decision  about  including  game -based  training  in  AtN. 
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ASSESSING  THE  USE  OF  GAME-BASED  EXERCISES  IN  THE  STAFF 
ATTACK-THE-NETWORK  COURSE 


Introduction 

The  U.S.  Army  soldier  trains  in  a  variety  of  learning  environments  (e.g.,  Live,  Virtual, 
and  Constructive).  Previous  and  ongoing  lessons  learned  from  the  Contemporary  Operational 
Environment  provides  relevance  to  planning,  rehearsing,  and  practicing  decision-making  on 
topics  related  to  mission  success.  Over  the  course  of  several  years,  the  U.S.  Army  increasingly 
relies  on  serious  games  as  a  training  tool  for  direct-fire  and  indirect-fire  tactical  tasks.  Although 
the  use  of  game-based  exercises  is  not  new  for  schoolhouse  courses  or  unit-based  training  on 
combat  tactics,  techniques,  and  procedures,  the  use  of  game -based  exercises  is  nonetheless  a 
novel  approach  for  training  human- intelligence  tasks.  Human-intelligence  tasks  are  the  actions 
related  to  collecting  information  from  people  and  other  sources  (i.e.,  social  networks,  print,  and 
visual  media)  to  identify  elements,  intentions,  composition,  strength,  dispositions,  tactics, 
equipment,  personnel,  and  capabilities  (Department  of  the  Army,  2011).  Currently,  these  skills 
are  trained  in  the  U.S.  Army’s  Staff  Attack  the  Network  (AtN)  course. 

The  question  remains,  however,  whether  or  not  game-based  training  environments  are 
effective  for  applying  human-intelligence  skills.  In  order  to  determine  the  extent  to  which  game- 
based  training  provided  an  effective  means  to  apply  human-intelligence  skill,  two  forms  of 
scenario-based  practical  exercises  were  compared  in  the  AtN  course.  Student  performance  on 
traditional  paper-based  practical  exercises  was  compared  to  performance  on  practical  exercises 
based  in  the  Army’s  Enhanced  Dynamic  Geo-Social  Environment  (EDGE)  game.  This  research 
was  conducted  in  support  of  the  U.S.  Army  Maneuver  Center  of  Excellence  Directorate  of 
Training  and  Doctrine  to  determine  possible  enhancements  to  the  AtN  course.  The  present 
report  describes  results  of  this  comparison  and  the  implications  for  training  human-intelligence 
skills  and  game-based  training. 

Games  Used  in  Military  Training 

Games  have  long  been  used  to  train  military  decision  making.  The  first  widely-known 
military  decision-making  game  was  introduced  in  1644,  and  simulations  of  unit  size,  terrain, 
visualization  of  movement  and  actions,  and  rate  of  individual  or  unit  speed  appeared  in  the  early 
1800s  (Gray,  1995).  Since  that  time,  complex  decision-making  simulations  have  been  used  to 
analyze  and  train  the  components  of  organizational  operations.  Game-based  training  offers  a 
low-cost  alternative  to  full  simulation  or  constructive  training  environments  (e.g.,  Ratwani, 

Orvis,  &  Knerr,  2010).  Game-based  training  also  facilitates  self-paced  or  individually-adaptive 
training  (Blankenbeckler,  Graves,  &  Wampler,  2013). 

Even  with  the  long  history  of  training  games,  few  evaluations  of  instructional 
effectiveness  were  conducted  before  the  late  1980’s  (Hays,  2005).  As  commercial  off-the-shelf 
video  game  technologies  were  applied  to  training  games,  more  research  on  training  and 
assessment  of  military  decision-making  games  has  emerged.  It  is  fairly  well  documented  that 
game-based  training  increases  motivation  to  train  during  game  play  (Dickey,  2007;  Ryan,  Rigby 
&  Przybylski,  2006;  Vogel,  Greenwood-Ericksen,  Cannon-Bowers,  &  Bowers,  2006)  and  that 
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game-based  training  is  pereeived  as  ehallenging  (Beal,  2007;  Singer,  Long,  Stahl,  &  Kusumoto, 
2008).  Moreover,  demonstrations  of  effective  game-based  training  in  military  settings  exist  when 
games  are  used  as  primary  instruction  (Pleban  &  Salter,  2001;  Ratwani,  et  ah,  2010)  or  as 
supplemental  instruction  (Topolsky,  et  ah,  2010). 

However,  it  is  not  clear  if  the  apparent  benefits  of  game-based  training  translate  to  actual 
increases  in  learning  or  performance  compared  to  traditional  modes  of  training  (Topolski  et  ah, 
2010).  Most  research  on  game-based  training  effectiveness  simply  analyzes  the  increase  in 
performance  from  a  baseline  (i.e.,  absolute  effectiveness)  as  opposed  analyzing  the  increase  in 
performance  compared  to  another  mode  of  training  (i.e.,  relative  effectiveness).  For  example, 
there  are  few  results  that  show  that  game-based  training  provided  larger  increases  in  training 
performance  or  retention  than  direct  instruction  or  paper-based  exercises  (Hays,  2005).  There  is 
also  little  evidence  to  suggest  that  game-based  training  transfers  to  actual  task  execution  (Hays; 
Temby,  Stephens,  &  Whitney,  2008). 

The  lack  of  research  on  relative  effectiveness,  coupled  with  the  lack  of  evidence  of 
transfer  and  with  the  ever-present  confounding  factors  described  in  the  research  literature,  makes 
it  difficult  to  determine  if  game-based  training  would  be  effective  for  human-intelligence  skills. 
In  addition  to  increasing  training  motivation,  game-based  training  has  been  shown  to  provide 
additional  complexity  to  training  (Singer,  et  ah,  2008;  Wolfe,  1997)  and  higher  operational 
relevance  (Lampton,  Clark,  &  Knerr,  2003)  compared  to  traditional  modes  of  training  (e.g., 
paper-based  exercises).  Because  human-intelligence  skills  require  decisions  in  complex  and 
ambiguous  situations,  game-based  training  may  be  especially  suited  for  use  in  AtN.  Moreover,  a 
consistent  contributing  factor  to  game-based  training  effectiveness  is  instructor/leader  facilitation 
of  the  training  (Beal,  2009;  Pleban  &  Salter,  2001;  Ratwani,  et  ah,  2010),  which  is  a  salient 
characteristic  of  AtN  training. 

Attack  the  Network  Course 

AtN  is  a  1-3  day  traditional  classroom  course  of  instruction  that  introduces  the 
fundamentals  of  human  intelligence  collection  to  battle  staff  personnel  and  their  supporting 
Company  Intelligence  Support  Team  (CoIST).  The  course  focuses  on  mission  planning  for 
targeting  social  network  activities.  The  targeting  principles  taught  in  the  AtN  course  applies  to 
intelligence  collections  of  various  social  networks  (i.e.,  criminal,  friendly,  and  belligerent)  with 
the  end  state  of  disrupting  enemy  network  operations.  Utilizing  ten  blocks  of  instructions. 
Soldiers  are  introduced  to  methodologies  and  lethal/non-lethal  targeting  methods  to  defeat, 
enable,  or  influence  networks  in  the  Area  of  Operations  (AO).  Traditionally,  a  series  of  practical 
exercises  using  paper-based  scenarios  provide  an  opportunity  for  Soldiers  to  apply,  analyze,  and 
synthesize  network-engagement  information  received  during  the  class  lecture.  As  a  comparison 
for  the  paper-based  exercises,  the  EDGE  game  was  introduced  to  the  AtN  course. 

Enhanced  Dynamic  Geo-Social  Environment 

EDGE  utilizes  a  commercial  game  engine  as  well  as  traditional  simulation  and  game- 
industry  standards  to  produce  a  training  platform  (Dwyer,  Griffith  &  Maxwell,  2011).  EDGE 
allows  players  to  control  an  in-game  avatar  (see  Figure  1),  navigate  terrain,  and  use  vehicles. 
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equipment,  and  tools  for  a  variety  of  social-cultural  task  and  other  actions  (see  Figure  2).  During 
a  capabilities  demonstration  in  2013,  first  responders  and  other  emergency  workers  (e.g.. 
Homeland  Security  and  dispatchers)  used  EDGE  to  train  in  complex  situations.  The  platform 
allowed  first  responders  from  across  jurisdictions  to  navigate  geo-specific  terrain;  utilize  ultra- 
high  resolution  vehicles,  equipment,  and  tools  to  execute  on-site  practices  and  procedures. 


Figure  1.  EDGE  Player  Avatar. 


Figure  2.  EDGE  Operational  Environment. 


Method 

The  basic  methodology  consisted  of  collecting  pre-test  and  post-test  knowledge  scores 
and  questionnaire  responses  from  Soldiers  completing  the  AtN  course  using  either  traditional 
paper-based  practical  exercises  (i.e..  Control  Group)  or  EDGE-based  practical  exercises  (i.e., 
EDGE  Group).  The  classroom  instruction  was  the  same  for  all  participants  and  the  only  variation 
in  the  course  was  the  conduct  of  the  practical  exercises.  AtN  is  a  mobile-training  course 
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provided  to  units  preparing  to  deploy.  As  sueh,  there  is  wide  variation  in  eourse  loeations  and  in 
the  military  oeeupation  specialties  of  the  students.  Every  effort  was  made  to  balance  this 
variability  across  the  experimental  conditions  (i.e.,  Control  Group  or  EDGE  Group). 

Participants 

Data  from  193  participants  was  collected  from  eight  AtN  classes  at  seven  Army 
installations.  The  installations  were  a  combination  of  active  component,  reserve  component,  and 
National  Guard  in  both  the  contiguous  United  States  and  outside  the  contiguous  United  States. 
Each  class  was  assigned  to  one  of  two  experimental  conditions  (i.e..  Control  or  EDGE)  in  quasi¬ 
random  order.  Because  the  entire  class  participated  in  the  same  condition,  the  two  experimental 
groups  somewhat  differed  on  the  number  and  types  of  military  occupation  specialty  (see  Table). 
Eour  participants  failed  to  complete  both  the  pre-test  and  the  post-test.  The  data  for  those 
participants  were  removed,  and  the  effective  sample  sizes  were  100  for  the  Control  Group  and  89 
for  the  EDGE  Group. 

Table. 

Participant  Military  Occupation  Specialties. 


Control 

EDGE 

Military  Intelligence 

15 

30 

Infantry 

14 

15 

Military  Police 

37 

15 

Civil  Affairs 

24 

- 

Engineer 

- 

8 

Eield  Artillery 

- 

7 

Other 

13 

15 

Total 

103 

90 

Materials 

Pre-test  and  Post-test,  The  AtN  course  test  consisted  of  27  completion  (i.e.,  “fill-in-the- 
blank”)  items  for  six  problem-based  themes:  (1)  Understanding  the  mission;  (2)  Understanding 
the  Operational  Environment;  (3)  Understanding  Networks;  (4)  Organize  for  the  Eight  and 
Engage  the  Network;  (5)  Targeting;  and  (6)  Assessment  (e.g.,  was  End  State  achieved?).  The 
course  test  was  used  as  the  post-test,  and  an  alternate  form  of  the  course  test  was  developed  by 
course  managers  to  be  used  as  the  pre-test.  Both  tests  addressed  the  six  course  themes  and 
assessed  the  same  type  of  information  but  differed  in  specific  content.  Answers  for  each  test 
were  provided  on  formatted  worksheets.  The  tests  were  scored  by  AtN  course  instructors 
according  to  the  approved  course  scoring  rubric.  There  were  a  total  of  100  points  possible  on 
each  test. 
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Paper-Based  Practical  Exercise.  The  paper-based  AtN  Practieal  Exercise  consisted  of 
an  introduction  and  a  four-part  scenario-based  problem.  The  practical  exercises  were  completed 
in  small  groups.  Each  part  has  a  specific  goal  for  the  group  to  complete  (e.g.  determine  any 
critical  gaps  in  information)  that  corresponded  to  the  just-completed  block  of  instruction.  The 
scenario  for  each  part  was  provided  in  a  hand-out,  and  each  group  was  given  a  booklet  with 
information  about  the  AO  (i.e.,  and  intelligence  brief).  The  groups  were  required  to  justify  their 
decisions  for  each  part  during  back  briefs  to  the  commander  role-player  in  the  classroom. 

In  Part  I,  Operational  Environment  Analysis,  group  members  read  the  operational 
environment  (OE)  brief  and  Commander’s  intent  and  were  directed  to  pay  attention  to  the 
descriptions  of  the  operational  variables  and  civil  considerations  in  the  AO.  The  group  goal  was 
to  identify  five  key  facts  about  the  OE.  In  Part  II,  Pattern  Analysis,  a  24-hour  day  scenario 
provided  an  opportunity  to  capture  real-time  intelligence.  This  intelligence  was  used  to  develop  a 
pattern-analysis  tool  and  products  to  report  critical  incidents  (i.e.  improvised  explosive  devise 
[lED]  encounter).  The  goal  of  Pattern  Analysis  was  to  determine  any  critical  gaps  in  information 
and  to  determine  the  kind  of  information  one  would  request  from  an  enabler  (e.g.  Counter  lED). 

In  Part  III,  Network  Analysis,  group  members  collaborated  to  identify  required  actions  to 
complete  an  activity  model  by  monitoring  network  actors  (e.g.  bomb  makers,  planners,  and 
videographers)  and  network  locations  (e.g.  auto  shops  and  shopping  markets).  Using  a  completed 
activity  matrix,  groups  identified  1 1  character  interactions  and  documented  the  time  and  location 
of  the  interactions.  In  Part  IV,  Targeting  and  Assessment,  group  members  worked  together  to 
identify  friendly,  neutral,  and  threat  networks  in  the  AO  and  to  develop  a  plan  that  integrated 
lethal  and  non-lethal  actions  to  generate  direct  and  indirect  effects. 

EDGE  Practical  Exercise.  The  EDGE -based  Practical  Exercise  consisted  of  six  virtual- 
world  days  of  human-intelligence  activities  in  a  fictitious  country.  Students  were  divided  into 
small  groups  depending  on  class  size  and  remained  with  their  assigned  group  through  completion 
of  the  Practical  Exercise.  At  the  end  of  the  final  block  of  classroom  instruction,  the  entire  class 
was  introduced  to  the  EDGE  game  (e.g.,  using  action  menu  and  controls)  and  completed 
familiarization  with  the  action  menu.  After  the  introduction,  students  began  game-play  by 
entering  the  virtual  OE,  were  directed  to  reference  materials,  received  an  OE  brief  and  operations 
order  from  a  Commander  avatar,  and  were  directed  to  pay  attention  to  the  descriptions  of  the 
operational  variables  and  civil  considerations  in  the  AO.  The  goal  of  the  game  was  to  interact 
with  friendly,  neutral,  and  hostile  avatars  in  the  OE,  collect  and  disseminate  information  between 
group  members,  and  collaborate  on  decisions  to  support,  monitor,  or  neutralize  networks. 

Specific  task  challenges  and  decision-making  events  were  conducted  during  virtual  Days  1-6. 
On  virtual  Day  7,  each  group  received  a  virtual  After  Actions  Review  (AAR)  on  courses  of 
actions  and  the  first-order  effects  and  second-order  effects  of  their  decisions. 

AtN  Post-Training  Questionnaire.  The  AtN  Post-Training  Questionnaire  captured 
participants’  post-training  perceptions  of  the  AtN  practical  exercises.  The  questionnaire 
collected  background  information  and  contained  items  to  assess  the  participants’  knowledge  and 
experience  with  other  methods  of  decision-making,  problem-solving,  and  mission  analysis  and 
tactical  risk  analysis  (i.e..  Troop  Leading  Procedures).  The  questionnaire  also  contained  16 
items  to  capture  perceptions  of  how  the  practical  exercises  influenced  learning  of  specific  AtN 
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skills  and  of  overall  understanding  of  AtN  concepts.  The  perception  items  were  rated  on  a  4- 
point  Likert-type  scale  with  anchors  at  “strongly  agree”  and  “strongly  disagree.”  The  AtN  Post- 
Training  Questionnaire  is  provided  in  Appendix  A. 

Procedure 

The  pre-test  was  administered  prior  to  classroom  instruction  to  assess  Soldiers’  existing 
level  of  knowledge  on  conducting  AtN -related  activities.  Soldiers  were  given  45  minutes  to 
complete  the  pre-test.  Classroom  instruction  for  all  students  included  ten  blocks  of  instructions. 
Each  block  of  instruction  lasted  approximately  two-hours  and  introduced  students  to 
methodologies  and  lethal/non-lethal  targeting  methods  to  defeat,  enable,  or  influence  networks  in 
an  AO.  Participants  in  the  Control  Group  (i.e.,  paper-based  practical  exercises)  were  given  a 
practical  exercise  at  the  end  of  each  day  of  training  and  completed  the  practical  exercises  as 
previously  described.  Participants  in  the  EDGE  Group  were  given  the  practical  exercise  after  the 
final  block  of  training  on  the  last  training  day  and  completed  the  practical  exercise  as  previously 
described.  This  procedural  confound  will  be  addressed  in  the  Discussion  section. 

After  the  completion  of  all  practical  exercises,  students  were  given  the  post-test.  Again, 
students  were  given  45  minutes  to  complete  this  test.  Once  the  post-test  was  completed,  the  AtN 
Post-Training  Questionnaire  was  administered.  Students  were  given  as  much  time  as  necessary 
to  complete  the  questionnaire,  and  most  students  finished  the  questionnaire  in  about  20  minutes. 
Einally,  participants  were  debriefed  about  the  purposes  of  the  research  and  the  final 
administrative  tasks  for  the  course  were  completed. 


Results 

The  relative  effectiveness  of  game-based  exercises  in  the  AtN  course  was  determined  by 
comparing  the  differences  between  pre-test  scores  and  post-test  scores  across  the  two 
experimental  conditions  (i.e..  Control  group  and  EDGE  Group).  If  game -based  exercises 
provided  additional  benefit  to  the  AtN  course,  then  the  EDGE  Group  should  have  a  greater 
increase  in  scores  than  the  Control  Group.  In  addition  to  the  analysis  of  test  scores,  training 
effectiveness  was  determined  from  questionnaire  responses  on  students’  perceptions  of  the 
practical  exercises.  In  this  case,  training  effectiveness  was  determined  (a)  by  affirming 
responses  to  the  questionnaire  items  and  (b)  by  differences  in  the  frequencies  of  responses 
between  the  two  experimental  conditions.  If  game-based  training  was  perceived  as  effective  in 
the  AtN  course,  then  the  EDGE  Group  should  both  have  “positive”  responses  to  questionnaire 
items  and  have  a  higher  frequency  of  positive  responses  than  the  Control  Group.  Einally,  an 
analysis  of  student  characteristics  was  conducted  to  see  if  differences  in  background  variables 
could  account  for  any  differences  in  post-test  scores.  This  analysis  could  (a)  further  illuminate 
the  nature  of  any  relative  effects  of  game -based  training  in  the  AtN  course  and  (b)  add  to  the 
evidence  that  describes  how  game-based  training  is  best  applied  (e.g.,  Topolski,  2010).  Each 
analysis  was  conducted  at  the  .05  level  of  alpha  error.  Appropriate  post-hoc  analyses  are  also 
reported. 
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Pre-test  and  Post-test  Scores 

The  percent  correct  on  each  AtN  course  test  was  compared  in  a  2  (pre-test,  post-test)  X  2 
(Control  Group,  EDGE  Group)  mixed-factors  ANOVA.  Overall,  test  scores  increased  nearly 
35%  (F(l,  187)  =  636.76,  MSe  =  .02, p  <  .01),  but  there  was  no  difference  in  test-score  increase 
between  the  experimental  conditions  (F(l,  187)  =  2.87,  MSe  =  .02,  p  =  .09).  Eigure  3  presents  the 
percent  correct  for  each  test  across  experimental  groups.  The  results  indicated  that  the  course 
and  practical  exercises  were  effective  at  increasing  human-intelligence  skills.  However,  the 
game-based  training  did  not  have  unique  effect  on  the  improvement  of  skill. 


Test 


■  Control 
EDGE 


Figure  3.  Percent  Correct  for  Each  AtN  Course  Test  across  Experimental  Groups.  Error  bars 
represent  95%  confidence  intervals. 


Training  Effectiveness  Items. 


On  the  AtN  Post-Training  Questionnaire,  students  indicated  the  level  of  agreement  to  16 
statements  about  the  impact  of  the  practical  exercises  on  learning  outcomes.  The  response 
frequencies  for  each  statement  were  analyzed  with  chi-square  to  determine  (a)  the  perception  of 
the  impact  of  the  practical  exercises  on  training  effectiveness  and  (b)  experimental  group 
differences  in  those  perceptions.  The  response  frequencies  for  all  16  statements  and  the 
statistical  results  for  the  analyses  of  the  frequencies  are  given  in  the  Appendix  B.  An  inspection 
of  Appendix  B  clearly  shows  that  students  overwhelmingly  “agreed”  (or  “strongly  agreed”)  with 
all  16  statements.  The  high  level  of  “agree”  responses  indicated  that  the  practical  exercises  were 
perceived  as  effective  for  training. 


However,  the  pattern  of  responses  differed  between  the  two  groups  for  six  of  the  16 
statements.  Eor  statements  that  differed,  the  Control  Group  had  more  “agree”  and  “strongly 
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agree”  responses  and  fewer  “disagree”  and  “strongly  disagree”  response  than  the  EDGE  Group. 
An  example  of  this  pattern  of  results  ean  be  seen  in  Eigure  4,  which  presents  the  percent  of 
responses  for  the  statement,  “The  Practical  Exercise  provided  challenging  experiences.”  The 
other  statements  that  followed  this  general  pattern  were: 

•  The  Practical  Exercise  had  a  valuable  impact  on  my  decision-making  skill. 

•  The  Practical  Exercise  helped  me  focus  on  Attack  the  Network  mission  concepts. 

•  The  Practical  Exercise  helped  me  understand  information  collection  and  Operational 
Environment  analyses. 

•  The  Practical  Exercise  was  a  valuable  learning  experience. 

•  The  Practical  Exercise  allowed  me  to  practice  measuring  Attack  the  Network  level  of 
effectiveness. 

100%  ^ 

90% - 

80% - 


© 

f  60% 

£  50% 

I  40% 

I  30% 

^  20% 

10% 

0% 

Resi>onse.s 

Figure  4.  Percent  of  Responses  to  the  Statement,  “The  Practical  Exercise  provided  challenging 
experiences”  across  Experimental  Groups. 

As  with  the  results  for  the  AtN  course  tests,  the  results  of  students’  perceptions  of  the 
practical  exercises  indicated  that  the  training  was  effective  but  there  was  little  difference  in 
effectiveness  between  experimental  groups.  The  few  group  differences  that  emerged  in  the 
students’  perceptions  data,  however,  appeared  to  favor  the  paper-based  exercises  as  the  more 
effective  training.  The  reason  for  the  more  favorable  perception  of  paper-based  exercises  was 
unclear.  One  speculation  was  that  the  paper-based  exercises  required  more  instructor-student 
and  student-student  interactions.  However,  a  more  careful  examination  of  participant 
characteristics  may  also  indicate  why  the  paper-based  exercises  were  perceived  as  being 
effective. 

Participant  Characteristics  Items 


I  Control 
EDGE 


Strongly  Disagree  Agree  Strongly 
Disagree  Agree 
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Additional  self-report  items  on  the  ATN  Post-Training  Questionnaire  (see  Appendix  A) 
were  eonsidered  as  eontributing  faetors  to  the  scores  on  the  AtN  course  post-test.  Those  items 
assessed  factors  such  as  formal  military  decision  making  experience  and  digital-technology 
experience  and  confidence.  The  only  group  difference  found  among  the  responses  for  these 
questionnaire  items  was  experience  with  Intelligence  Preparation  of  the  Battlefield  (IPB) 
process.  Students  in  the  EDGE  Group  self-reported  more  experience  with  IPB  than  the  Control 
Group  (x^  (3)  =  8.80, <  .05).  However,  an  examination  of  Eigure  5  reveals  that  this  group 
difference  was  primarily  driven  by  differences  in  frequencies  for  “little  experience”  and  “some 
experience”  responses.  As  such,  it  did  not  appear  that  the  difference  in  experience  with  IPB  was 
overwhelming.  In  addition  to  the  lack  of  group  differences  across  most  items,  no  items  produced 
statistically  significant  correlations  with  post-test  scores  (highest  r  =  .14).  Thus,  it  did  not  appear 
that  participant  characteristics  could  account  for  the  patterns  of  results  seen  in  the  AtN  post-test 
scores. 
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Figure  5.  Percent  of  Responses  for  “Experience  with  using  Intelligence  Preparation  of  the 
Battlefield  Process”  across  Experimental  Groups. 


Discussion 

The  purpose  of  the  research  reported  here  was  to  investigate  the  extent  to  which  game- 
based  training  could  be  used  to  enhance  the  training  of  human- intelligence  skills.  In  this  case, 
EDGE  was  used  to  deliver  practical  exercises  as  part  of  the  Army’s  AtN  course.  The  game- 
based  (i.e.,  EDGE)  practical  exercises  did  no  better  in  increasing  end-of-course  test  scores  than 
did  traditional  paper-based  practical  exercises.  In  addition,  the  paper-based  practical  exercises 
were  perceived  as  more  beneficial  to  some  learning  and  course  outcomes  than  the  game-based 
practical  exercises.  These  results  add  to  the  growing  literature  that  fails  to  find  a  relative 
advantage  of  game-based  training  (e.g.,  Topolski,  et  ah,  2010)  and  highlight  the  need  for  more 
controlled  comparisons  of  game-based  training  effectiveness. 
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Clearly,  there  was  no  advantage  for  game-based  training  in  the  present  comparison. 
However,  caution  should  be  taken  in  interpreting  these  results  because  of  procedural  differences 
in  the  two  training  conditions.  Students  who  used  paper-based  practical  exercises  completed  the 
exercises  after  each  phase  of  instruction,  while  students  who  used  the  EDGE  practical  exercises 
did  so  only  at  the  completion  of  classroom  instruction.  That  is,  the  paper-based  practical 
exercises  were  spaced  across  sessions,  whereas  the  EDGE  practical  exercises  were  massed  into 
one  session.  It  may  be  the  case  that  the  students  who  used  paper-based  practical  exercises 
received  a  distributed-practice  benefit  to  decision-making  skills  (Crowder,  1976;  Taylor  & 

Roher,  2010). 

Another  procedural  difference  between  the  two  training  conditions  was  the  level  of 
instructor  involvement  during  the  practical  exercises.  In  the  EDGE  practical  exercises,  the  game 
guided  the  flow  of  information  and  nature  of  student  interactions  with  minimal  instructor 
involvement.  By  contrast,  the  paper-based  practical  exercises  required  instructor  guidance,  and 
the  instructor  was  involved  with  answering  students’  questions.  As  such,  students  who  used  the 
paper-based  practical  exercises  may  have  received  a  benefit  from  instructor  interaction 
(Thurmond  &  Wambach,  2004). 

Both  of  these  procedural  differences  (i.e.,  distributed  practice  and  instructor  interaction) 
should  have  benefited  students  who  used  paper-based  practical  exercises.  It  might  be  argued 
then  that  the  effects  of  the  procedural  differences  offset  the  advantage  of  game-based  training, 
which  led  to  equivalent  end-of-course  test  performance  across  the  two  training  conditions.  In  the 
same  vein,  it  might  be  argued  that  the  procedural  differences  contributed  to  the  more  favorable 
perceptions  of  effectiveness  for  the  paper-based  practical  exercises.  However,  it  was  equally 
likely  that  the  procedural  differences  had  no  effect  on  test  performance  or  perceptions  of  training 
effectiveness  and  that  game-based  training  simply  provided  no  advantage  for  decision-making 
skills.  Eikewise,  it  was  possible  that  neither  type  of  practical  exercise  had  any  effect  on  test 
performance  but  did  influence  perceptions  of  training.  That  is,  it  may  be  the  case  that  the 
increase  in  test  scores  was  a  result  of  the  course  instruction  and  not  the  practical  exercises. 

Obviously,  the  generality  of  the  current  results  should  not  be  overstated,  and  additional 
empirical  research  is  needed  to  further  clarify  the  potential  effects  noted  here.  The  level  of 
instructor  interaction  would  be  a  logical  factor  to  compare  between  game-based  training  and  the 
traditional  training  environment.  A  similar  research  direction  would  be  to  explore  the  efficacy  of 
in-game  resources  (e.g.,  pull-down  menus,  non-player  avatars,  and  help  functions)  on  training 
outcomes.  In-game  resources  provide  similar  types  of  information  and  guidance  as  instructors 
but  are  always  available.  By  contrast,  the  in-game  resources  may  not  always  have  the  high-level 
understanding  of  the  skills  or  the  ability  to  adapt  responses  to  learners  needs. 

Because  the  results  presented  here  were  not  conclusive,  it  should  be  noted  that  game- 
based  training  may  still  hold  promise  for  training  human-intelligence  skills.  EDGE  supported 
several  functions  critical  for  developing  human-intelligence  skills.  Eirst,  EDGE  provided 
performance-feedback  on  decisions  made  while  in  play,  which  is  critical  for  learning  in  virtual 
environments  (Blaiwes,  Puig,  &  Regan,  1973;  Bransford,  Brown,  &  Cocking,  2000).  Second, 
the  simulated  multi-day  scenario  allowed  periods  of  reflection  and  opportunities  for  self-learning 
(Bringle  &  Hatcher,  1999)  that  were  not  available  in  the  paper-based  exercises.  Third,  EDGE 
provided  opportunities  for  unscripted  interactions  with  host-nation  citizens  that  were  not  possible 
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in  the  paper-based  exercise.  Finally,  the  visual  characteristics  of  the  virtual  environment  allow 
students  to  develop  information  from  non-verbal  cues  (Hinde,  1972;  Knapp,  Hall,  &  Morgan, 
2013). 


The  Army  Learning  Model  outlines  a  set  of  soldier  competencies  geared  toward  21st 
century  missions  (Department  of  the  Army,  2011).  These  competencies  emphasize  the  need  for 
Soldiers  to  develop  skills  in  nontraditional  areas,  such  as  moral-ethical  decision-making,  social- 
cultural  awareness,  and  cognitive  reasoning.  The  Army  Leaning  Model  recognizes  the  need  to 
train  these  competencies  in  relevant  and  challenging  contexts  and  at  the  point  of  need.  Game- 
based  training  holds  the  promise  to  meet  the  requirements  of  the  Army  Learning  Model,  but 
additional  conceptual  development  of  game  structure  and  employment  may  be  required  before 
that  promise  is  met.  The  findings  from  the  current  research  were  briefed  to  the  Maneuver  Center 
of  Excellence  Directorate  of  Training  and  Doctrine  and  to  the  U.S.  Army  Research  Laboratory  - 
Human  Research  and  Engineering  Directorate  Simulation  and  Training  Technology  Center.  The 
findings  were  used  to  inform  decision  about  including  game-based  training  in  the  AtN  course. 
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APPENDIX  A 

AtN  Post-Training  Questionnaire 


A-1 


Training  Experience  Questionnaire 


Instructions:  In  an  effort  for  continuous  improvement  of  training,  the  Maneuver  Center  of 
Exeellence  would  like  to  have  your  input  on  your  experiences  in  this  course.  For  each  item, 
please  provide  your  honest  answer.  Your  participation  in  this  questionnaire  is  voluntary.  You 
may  stop  responding  at  any  time  and  submit  the  questionnaire  to  the  designated  person  without 
consequence. 


1 .  Rank _ 

2.  MOS/Branch _ 

3.  Time  in  service:  Years _ Months _ 

4.  How  long  have  you  been  in  your  current  unit?  Y ears _ Months 

5.  What  is  your  current  duty  position? _ 

6.  How  many  months  have  you  been  in  this  position? _ 

7.  Please  list  any  battalion  or  brigade  staff  experience  you  have. 


8.  Do  you  have  combat  theater  experience?  No _ Yes _ 

9.  How  many  combat  theater  deployments  have  you  had? _ 

10.  What  is  your  level  of  confidence  in  using  digital  technology  (i.e.  Personal  Computer,  Smartphone, 
or  Tablet)?  (Mark  one  only) 


None 

Low 

Some 

High 

□ 

□ 

□ 

□ 

1 1 .  How  many  hours  per  week  do  you  play  tactical  (i.e.  first  person)  or  strategic  (i.e.  turn-based) 
computer/video  games?  (Mark  one  only) 


0  -  4hours 

5-8  hours 

9-12  hours 

12  hours  + 

□ 

□ 

□ 

□ 

A-2 


For  each  of  the  following  tasks,  please  indicate  your  level  of 
performance  experience. 

Very 

Experience 

Some 

Experience 

Little 

Experience 

No 

Experience 

1.  Applying  METT-TC 

□ 

□ 

□ 

□ 

2.  Applying  OAKOC 

□ 

□ 

□ 

□ 

3.  Combat  Profiling 

□ 

□ 

□ 

□ 

4.  Considering  Combat  Multipliers 

□ 

□ 

□ 

□ 

5.  IPB  Process 

□ 

□ 

□ 

□ 

6.  Making  informed  decisions  to  conserve  lives  and  resources  to 
avoid  unnecessary  risk 

□ 

□ 

□ 

□ 

7.  Making  informed  decisions  based  on  implemented  courses  of 
action 

□ 

□ 

□ 

□ 

8.  Identifying  feasible  and  effective  control  measures  in  the  absence 
of  specific  standards 

□ 

□ 

□ 

□ 

9.  Develop  reasonable  alternatives  to  accomplish  mission 

□ 

□ 

□ 

□ 

A-3 


The  following  items  refer  to  your  experience  with  the  course  Practical 
Exercises  that  you  completed.  For  each  of  the  statements,  please 
indicate  the  extent  to  which  you  agree  or  disagree. 

Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

1 .  The  Practical  Exercises  provided  me  an  opportunity  to  practice 
making  sound  tactical  decisions. 

□ 

□ 

□ 

□ 

2.  The  Practical  Exercises  improved  my  ability  to  make  more  rapid 
tactical  decisions. 

□ 

□ 

□ 

□ 

3.  The  Practical  Exercises  made  me  more  confident  in  my  ability  to 
make  tactical  decisions. 

□ 

□ 

□ 

□ 

4.  The  Practical  Exercises  provided  challenging  experiences. 

□ 

□ 

□ 

□ 

5.  The  Practical  Exercises  had  a  valuable  impact  on  my  decision¬ 
making  skills. 

□ 

□ 

□ 

□ 

6.  The  Practical  Exercises  helped  me  focus  on  Attack  the  Network 
mission  concepts. 

□ 

□ 

□ 

□ 

7.  The  Practical  Exercises  helped  me  understand  information  collection 
and  Operational  Environment  analyses. 

□ 

□ 

□ 

□ 

8.  The  Practical  Exercises  allowed  me  to  practice  the  types  of  decisions 

1  must  make  as  a  small  unit  leader. 

□ 

□ 

□ 

□ 

9.  The  Practical  Exercises  were  a  valuable  learning  experience. 

□ 

□ 

□ 

□ 

10.  The  Practical  Exercises  allowed  me  to  practice  assessing  needs  and 
requesting  enablers  to  engage  a  network. 

□ 

□ 

□ 

□ 

1 1 .  The  Practical  Exercises  allowed  me  to  practice  interacting  with  host 
nation  citizens. 

□ 

□ 

□ 

□ 

12.  The  Practical  Exercises  allowed  me  to  practice  collaborating  with 
intergovernmental  organizations,  non-govemmental  organizations, 
or  other  agencies  to  engage  a  network. 

□ 

□ 

□ 

□ 

13.  The  Practical  Exercises  improved  my  ability  to  determine  strengths 
and  weaknesses  of  a  network. 

□ 

□ 

□ 

□ 

14.  The  Practical  Exercises  made  me  more  confident  in  my  ability  to 
identify  High  Payoff  Targets  (HPT’s). 

□ 

□ 

□ 

□ 

15.  The  Practical  Exercises  helped  me  determine  best  targeting  method 
to  employ  against  HPTs. 

□ 

□ 

□ 

□ 

16.  The  Practical  Exercises  allowed  me  to  practice  measuring  Attack  the 
Network  level  of  effectiveness. 

□ 

□ 

□ 

□ 

Thank  you  for  your  Participation, 
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APPENDIX  B 

Response  Frequencies  and  Test  Statistics  for  Training  Effectiveness  Items 


B-1 


Response  frequencies  for  each  experimental  group  on  AtN  Post-Training  Questionnaire 
statements  of  practical  exercise  training  effectiveness.  The  test  statistie  (i.e.,  chi  square)  is 
provided.  Statistieally  signifieant  differenees  in  response  frequeneies  between  experimental 
groups  (i.e.,  Control  and  EDGE)  at  the  alpha  =  .05  level  are  indieated  by  statements  in  bold  type 
and  asterisks  next  to  the  test  statistic. 


The  Practical  Exercises  provided  me  an  opportunity  to  practice  making  sound  tactical 
decisions. 


Strongly 

Strongly 

Disagree  Disagree 

Agree 

Agree 

Control 

0  2 

56 

42 

1  ^  ^ 

EDGE 

2  4 

50 

33 

X  =  3.46 

The  Practical  Exercises  improved  my  ability  to  make  i 

more  rapid  tactical  decisions. 

Strongly 

Strongly 

Disagree  Disagree 

Agree 

Agree 

Control 

1  12 

54 

33 

EDGE 

3  17 

47 

22 

X'  =  3.92 

The  Practical  Exercises  made  me  more  confident  in  my  ability  to  make  tactical  decisions. 

Strongly 

Strongly 

Disagree  Disagree 

Agree 

Agree 

Control 

0  10 

54 

36 

0  .  ^  - 

EDGE 

3  11 

51 

24 

X  =  4.91 

The  Practical  Exercises  provided  challenging  experiences. 

Strongly 

Strongly 

Disagree  Disagree 

Agree 

Agree 

Control 

0  3 

49 

48 

1  ^  ^  ^ 

EDGE 

2  13 

51 

23 

X  =  16.51 

The  Practical  Exercises  had  a 

valuable  impact  on  my  decision-making  skills. 

Strongly 

Strongly 

Disagree  Disagree 

Agree 

Agree 

Control 

0  5 

55 

40 

1  ^  ^ 

EDGE 

2  12 

56 

19 
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The  Practical  Exercises  helped  me  focus  on  Attack  the  Network  mission  concepts. 


Strongly 

Strongly 

Disagree  Disagree 

Agree 

Agree 

Control 

1  2 

43 

54 

EDGE 

3  7 

54 

25 

X^=  15.08* 

The  Practical  Exercises  helped  me  understand  information  collection  and  Operational 

Environment  analyses. 

Strongly 

Strongly 

Disagree  Disagree 

Agree 

Agree 

Control 

0  5 

43 

52 

^  =  9.06* 

EDGE 

1  5 

55 

28 

The  Practical  Exercises  allowed  me  to  practice  the  types  of  decisions  I  must  make  as  a  small 

unit  leader. 

Strongly 

Strongly 

Disagree  Disagree 

Agree 

Agree 

Control 

1  8 

53 

38 

EDGE 

1  9 

59 

20 

X'  =  5.34 

The  Practical  Exercises  was  a 

valuable  learning  experience. 

Strongly 

Strongly 

Disagree  Disagree 

Agree 

Agree 

Control 

0  2 

41 

57 

EDGE 

1  7 

46 

34 

X'  =  9.15* 

The  Practical  Exercises  allowed  me  to  practice  assessing  needs  and  requesting  enablers  to 

engage  a  network. 

Strongly 

Strongly 

Disagree  Disagree 

Agree 

Agree 

Control 

1  10 

51 

38 

EDGE 

0  8 

54 

27 

X'  =  2.54 

The  Practical  Exercises  allowed  me  to  practice  interacting  with  host  nation  citizens. 

Strongly 

Strongly 

Disagree  Disagree 

Agree 

Agree 

Control 

8  22 

48 

22 

9  _ _ 

EDGE 

2  12 

49 

25 

X  =  6.00 
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The  Practical  Exercises  allowed  me  to  practice  collaborating  with  intergovernmental 
organizations,  non-governmental  organizations,  or  other  agencies  to  engage  a  network. 


Strongly 

Strongly 

Disagree 

Disagree 

Agree 

Agree 

Control 

5 

29 

47 

19  ,  . 

EDGE 

3 

22 

46 

V 

W 

oo 

The  Practical  Exercises  improved  my  ability  to  determine  strengths  and  weaknesses  of  a 

network. 

Strongly 

Strongly 

Disagree 

Disagree 

Agree 

Agree 

Control 

2 

4 

55 

38  2 

EDGE 

1 

8 

60 

Y  =  6.96 

20  ^ 

The  Practical  Exercises  made  me  more  confident  in  my  ability  to  identify  High  Payoff 

Targets. 

Strongly 

Strongly 

Disagree 

Disagree 

Agree 

Agree 

Control 

2 

10 

49 

38  2  .  .. 

EDGE 

3 

13 

51 

X  =  4.38 

22  ^ 

The  Practical  Exercises  helped  me  determine  best  targeting  method  to  employ  against  High 

Payoff  Targets. 

Strongly 

Strongly 

Disagree 

Disagree 

Agree 

Agree 

Control 

1 

15 

50 

34  2  . 

EDGE 

2 

14 

56 

17  ^  ■ 

The  Practical  Exercises  allowed  me  to  practice  measuring  Attack  the  Network  level  of 

effectiveness 

Strongly 

Strongly 

Disagree 

Disagree 

Agree 

Agree 

Control 

0 

2 

60 

38  2  -tn.  ^Q,|, 

X  =  12.59* 

EDGE 

0 

13 

55 
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